Auditory Feature-based Perceptual Distance.

Studies comparing acoustic signals often rely on pixel-wise differences between spectrograms, as in for example mean squared error (MSE). Pixel-wise errors are not representative of perceptual sensitivity, however, and such measures can be highly sensitive to small local signal changes that may be imperceptible. In computer vision, high-level visual features extracted with convolutional neural networks (CNN) can be used to calculate the fidelity of computer-generated images. Here, we propose the auditory perceptual distance (APD) metric based on acoustic features extracted with an unsupervised CNN and validated by perceptual behavior. Using complex vocal signals from songbirds, we trained a Siamese CNN on a self-supervised task using spectrograms rescaled to match the auditory frequency sensitivity of European starlings, Sturnus vulgaris. We define APD for any pair of sounds as the cosine distance between corresponding feature vectors extracted by the trained CNN. We show that APD is more robust to temporal and spectral translation than MSE, and captures the sigmoidal shape of typical behavioral psychometric functions over complex acoustic spaces. When fine-tuned using starlings' behavioral judgments of naturalistic song syllables, the APD model yields even more accurate predictions of perceptual sensitivity, discrimination, and categorization on novel complex (high-dimensional) acoustic dimensions, including diverging decisions for identical stimuli following different training conditions. Thus, the APD model outperforms MSE in robustness and perceptual accuracy, and offers tunability to match experience-dependent perceptual biases.

The effects of range and echo-phase on range resolution in bottlenose dolphins (Tursiops truncatus) performing a successive comparison taska).

Echolocating bats and dolphins use biosonar to determine target range, but differences in range discrimination thresholds have been reported for the two species. Whether these differences represent a true difference in their sensory system capability is unknown. Here, the dolphin's range discrimination threshold as a function of absolute range and echo-phase was investigated. Using phantom echoes, the dolphins were trained to echo-inspect two simulated targets and indicate the closer target by pressing a paddle. One target was presented at a time, requiring the dolphin to hold the initial range in memory as they compared it to the second target. Range was simulated by manipulating echo-delay while the received echo levels, relative to the dolphins' clicks, were held constant. Range discrimination thresholds were determined at seven different ranges from 1.75 to 20 m. In contrast to bats, range discrimination thresholds increased from 4 to 75 cm, across the entire ranges tested. To investigate the acoustic features used more directly, discrimination thresholds were determined when the echo was given a random phase shift (±180°). Results for the constant-phase versus the random-phase echo were quantitatively similar, suggesting that dolphins used the envelope of the echo waveform to determine the difference in range.

Acoustic and language-specific sources for phonemic abstraction from speech.

Spoken language comprehension requires abstraction of linguistic information from speech, but the interaction between auditory and linguistic processing of speech remains poorly understood. Here, we investigate the nature of this abstraction using neural responses recorded intracranially while participants listened to conversational English speech. Capitalizing on multiple, language-specific patterns where phonological and acoustic information diverge, we demonstrate the causal efficacy of the phoneme as a unit of analysis and dissociate the unique contributions of phonemic and spectrographic information to neural responses. Quantitive higher-order response models also reveal that unique contributions of phonological information are carried in the covariance structure of the stimulus-response relationship. This suggests that linguistic abstraction is shaped by neurobiological mechanisms that involve integration across multiple spectro-temporal features and prior phonological information. These results link speech acoustics to phonology and morphosyntax, substantiating predictions about abstractness in linguistic theory and providing evidence for the acoustic features that support that abstraction.

Syntactic modulation of rhythm in Australian pied butcherbird song.

The acoustic structure of birdsong is spectrally and temporally complex. Temporal complexity is often investigated in a syntactic framework focusing on the statistical features of symbolic song sequences. Alternatively, temporal patterns can be investigated in a rhythmic framework that focuses on the relative timing between song elements. Here, we investigate the merits of combining both frameworks by integrating syntactic and rhythmic analyses of Australian pied butcherbird (Cracticus nigrogularis) songs, which exhibit organized syntax and diverse rhythms. We show that rhythms of the pied butcherbird song bouts in our sample are categorically organized and predictable by the song's first-order sequential syntax. These song rhythms remain categorically distributed and strongly associated with the first-order sequential syntax even after controlling for variance in note length, suggesting that the silent intervals between notes induce a rhythmic structure on note sequences. We discuss the implication of syntactic-rhythmic relations as a relevant feature of song complexity with respect to signals such as human speech and music, and advocate for a broader conception of song complexity that takes into account syntax, rhythm, and their interaction with other acoustic and perceptual features.

Long-range sequential dependencies precede complex syntactic production in language acquisition.

To convey meaning, human language relies on hierarchically organized, long-range relationships spanning words, phrases, sentences and discourse. As the distances between elements (e.g. phonemes, characters, words) in human language sequences increase, the strength of the long-range relationships between those elements decays following a power law. This power-law relationship has been attributed variously to long-range sequential organization present in human language syntax, semantics and discourse structure. However, non-linguistic behaviours in numerous phylogenetically distant species, ranging from humpback whale song to fruit fly motility, also demonstrate similar long-range statistical dependencies. Therefore, we hypothesized that long-range statistical dependencies in human speech may occur independently of linguistic structure. To test this hypothesis, we measured long-range dependencies in several speech corpora from children (aged 6 months-12 years). We find that adult-like power-law statistical dependencies are present in human vocalizations at the earliest detectable ages, prior to the production of complex linguistic structure. These linguistic structures cannot, therefore, be the sole cause of long-range statistical dependencies in language.

Parametric UMAP Embeddings for Representation and Semisupervised Learning.

UMAP is a nonparametric graph-based dimensionality reduction algorithm using applied Riemannian geometry and algebraic topology to find low-dimensional embeddings of structured data. The UMAP algorithm consists of two steps: (1) computing a graphical representation of a data set (fuzzy simplicial complex) and (2) through stochastic gradient descent, optimizing a low-dimensional embedding of the graph. Here, we extend the second step of UMAP to a parametric optimization over neural network weights, learning a parametric relationship between data and embedding. We first demonstrate that parametric UMAP performs comparably to its nonparametric counterpart while conferring the benefit of a learned parametric mapping (e.g., fast online embeddings for new data). We then explore UMAP as a regularization, constraining the latent distribution of autoencoders, parametrically varying global structure preservation, and improving classifier accuracy for semisupervised learning by capturing structure in unlabeled data.1.

Local field potentials in a pre-motor region predict learned vocal sequences.

Neuronal activity within the premotor region HVC is tightly synchronized to, and crucial for, the articulate production of learned song in birds. Characterizations of this neural activity detail patterns of sequential bursting in small, carefully identified subsets of neurons in the HVC population. The dynamics of HVC are well described by these characterizations, but have not been verified beyond this scale of measurement. There is a rich history of using local field potentials (LFP) to extract information about behavior that extends beyond the contribution of individual cells. These signals have the advantage of being stable over longer periods of time, and they have been used to study and decode human speech and other complex motor behaviors. Here we characterize LFP signals presumptively from the HVC of freely behaving male zebra finches during song production to determine if population activity may yield similar insights into the mechanisms underlying complex motor-vocal behavior. Following an initial observation that structured changes in the LFP were distinct to all vocalizations during song, we show that it is possible to extract time-varying features from multiple frequency bands to decode the identity of specific vocalization elements (syllables) and to predict their temporal onsets within the motif. This demonstrates the utility of LFP for studying vocal behavior in songbirds. Surprisingly, the time frequency structure of HVC LFP is qualitatively similar to well-established oscillations found in both human and non-human mammalian motor areas. This physiological similarity, despite distinct anatomical structures, may give insight into common computational principles for learning and/or generating complex motor-vocal behaviors.

Neurally driven synthesis of learned, complex vocalizations.

Brain machine interfaces (BMIs) hold promise to restore impaired motor function and serve as powerful tools to study learned motor skill. While limb-based motor prosthetic systems have leveraged nonhuman primates as an important animal model,1-4 speech prostheses lack a similar animal model and are more limited in terms of neural interface technology, brain coverage, and behavioral study design.5-7 Songbirds are an attractive model for learned complex vocal behavior. Birdsong shares a number of unique similarities with human speech,8-10 and its study has yielded general insight into multiple mechanisms and circuits behind learning, execution, and maintenance of vocal motor skill.11-18 In addition, the biomechanics of song production bear similarity to those of humans and some nonhuman primates.19-23 Here, we demonstrate a vocal synthesizer for birdsong, realized by mapping neural population activity recorded from electrode arrays implanted in the premotor nucleus HVC onto low-dimensional compressed representations of song, using simple computational methods that are implementable in real time. Using a generative biomechanical model of the vocal organ (syrinx) as the low-dimensional target for these mappings allows for the synthesis of vocalizations that match the bird's own song. These results provide proof of concept that high-dimensional, complex natural behaviors can be directly synthesized from ongoing neural activity. This may inspire similar approaches to prosthetics in other species by exploiting knowledge of the peripheral systems and the temporal structure of their output.

Toward a Computational Neuroethology of Vocal Communication: From Bioacoustics to Neurophysiology, Emerging Tools and Future Directions.

Recently developed methods in computational neuroethology have enabled increasingly detailed and comprehensive quantification of animal movements and behavioral kinematics. Vocal communication behavior is well poised for application of similar large-scale quantification methods in the service of physiological and ethological studies. This review describes emerging techniques that can be applied to acoustic and vocal communication signals with the goal of enabling study beyond a small number of model species. We review a range of modern computational methods for bioacoustics, signal processing, and brain-behavior mapping. Along with a discussion of recent advances and techniques, we include challenges and broader goals in establishing a framework for the computational neuroethology of vocal communication.

Spike Train Coactivity Encodes Learned Natural Stimulus Invariances in Songbird Auditory Cortex.

The capacity for sensory systems to encode relevant information that is invariant to many stimulus changes is central to normal, real-world, cognitive function. This invariance is thought to be reflected in the complex spatiotemporal activity patterns of neural populations, but our understanding of population-level representational invariance remains coarse. Applied topology is a promising tool to discover invariant structure in large datasets. Here, we use topological techniques to characterize and compare the spatiotemporal pattern of coactive spiking within populations of simultaneously recorded neurons in the secondary auditory region caudal medial neostriatum of European starlings (Sturnus vulgaris). We show that the pattern of population spike train coactivity carries stimulus-specific structure that is not reducible to that of individual neurons. We then introduce a topology-based similarity measure for population coactivity that is sensitive to invariant stimulus structure and show that this measure captures invariant neural representations tied to the learned relationships between natural vocalizations. This demonstrates one mechanism whereby emergent stimulus properties can be encoded in population activity, and shows the potential of applied topology for understanding invariant representations in neural populations.SIGNIFICANCE STATEMENT Information in neural populations is carried by the temporal patterns of spikes. We applied novel mathematical tools from the field of algebraic topology to quantify the structure of these temporal patterns. We found that, in a secondary auditory region of a songbird, these patterns reflected invariant information about a learned stimulus relationship. These results demonstrate that topology provides a novel approach for characterizing neural responses that is sensitive to invariant relationships that are critical for the perception of natural stimuli.

Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires.

Animals produce vocalizations that range in complexity from a single repeated call to hundreds of unique vocal elements patterned in sequences unfolding over hours. Characterizing complex vocalizations can require considerable effort and a deep intuition about each species' vocal behavior. Even with a great deal of experience, human characterizations of animal communication can be affected by human perceptual biases. We present a set of computational methods for projecting animal vocalizations into low dimensional latent representational spaces that are directly learned from the spectrograms of vocal signals. We apply these methods to diverse datasets from over 20 species, including humans, bats, songbirds, mice, cetaceans, and nonhuman primates. Latent projections uncover complex features of data in visually intuitive and quantifiable ways, enabling high-powered comparative analyses of vocal acoustics. We introduce methods for analyzing vocalizations as both discrete sequences and as continuous latent variables. Each method can be used to disentangle complex spectro-temporal structure and observe long-timescale organization in communication.

Epi-Intra neural probes with glassy carbon microelectrodes help elucidate neural coding and stimulus encoding in 3D volume of tissue.

OBJECTIVE: In this study, we demonstrate practical applications of a novel 3-dimensional neural probe for simultaneous electrophysiological recordings from the surface of the brain as well as deep intra-cortical tissue. We used this 3D probe to investigate signal propagation mechanisms between neuronal cells and their responses to stimuli in a 3D fashion. APPROACH: This novel probe leverage 2D thin-film microfabrication technique to combine an epi-cortical (surface) and an intra-cortical (depth) microelectrode arrays (Epi-Intra), that unfold into an origami 3D-like probe during brain implantation. The flexible epi-cortical component conforms to the brain surface while the intra-cortical array is reinforced with stiffer durimide polymer layer for ease of tissue penetration. The microelectrodes are made of glassy carbon material that is biocompatible and has low electrochemical impedance that is important for high fidelity neuronal recordings. These recordings were performed on the auditory region of anesthetized European starling songbirds during playback of conspecific songs as auditory stimuli. MAIN RESULTS: The Epi-Intra probe recorded broadband activity including local field potentials (LFPs) signals as well as single-unit activity and multi-unit activity from both surface and deep brain. The majority of recorded cellular activities were stimulus-locked and exhibited low noise. Notably, while LFPs recorded on surface and depth electrodes did not exhibit strong correlation, composite receptive fields (CRFs)-extracted from individual neuron cells through a non-linear model and that are cell-dependent-were correlated. SIGNIFICANCE: These findings demonstrate that CRFs extracted from Epi-Intra recordings are excellent candidates for neural coding and for understanding the relationship between sensory neuronal responses and their stimuli (stimulus encoding). Beyond CRFs, this novel neural probe may enable new spatiotemporal 3D volumetric mapping to address, with cellular resolution, how the brain coordinates function.

Stimulus Driven Single Unit Activity From Micro-Electrocorticography.

High-fidelity measurements of neural activity can enable advancements in our understanding of the neural basis of complex behaviors such as speech, audition, and language, and are critical for developing neural prostheses that address impairments to these abilities due to disease or injury. We develop a novel high resolution, thin-film micro-electrocorticography (micro-ECoG) array that enables high-fidelity surface measurements of neural activity from songbirds, a well-established animal model for studying speech behavior. With this device, we provide the first demonstration of sensory-evoked modulation of surface-recorded single unit responses. We establish that single unit activity is consistently sensed from micro-ECoG electrodes over the surface of sensorimotor nucleus HVC (used as a proper name) in anesthetized European starlings, and validate responses with correlated firing in single units recorded simultaneously at surface and depth. The results establish a platform for high-fidelity recording from the surface of subcortical structures that will accelerate neurophysiological studies, and development of novel electrode arrays and neural prostheses.

Five Ways in Which Computational Modeling Can Help Advance Cognitive Science: Lessons From Artificial Grammar Learning.

There is a rich tradition of building computational models in cognitive science, but modeling, theoretical, and experimental research are not as tightly integrated as they could be. In this paper, we show that computational techniques-even simple ones that are straightforward to use-can greatly facilitate designing, implementing, and analyzing experiments, and generally help lift research to a new level. We focus on the domain of artificial grammar learning, and we give five concrete examples in this domain for (a) formalizing and clarifying theories, (b) generating stimuli, (c) visualization, (d) model selection, and (e) exploring the hypothesis space.

Parallels in the sequential organization of birdsong and human speech.

Human speech possesses a rich hierarchical structure that allows for meaning to be altered by words spaced far apart in time. Conversely, the sequential structure of nonhuman communication is thought to follow non-hierarchical Markovian dynamics operating over only short distances. Here, we show that human speech and birdsong share a similar sequential structure indicative of both hierarchical and Markovian organization. We analyze the sequential dynamics of song from multiple songbird species and speech from multiple languages by modeling the information content of signals as a function of the sequential distance between vocal elements. Across short sequence-distances, an exponential decay dominates the information in speech and birdsong, consistent with underlying Markovian processes. At longer sequence-distances, the decay in information follows a power law, consistent with underlying hierarchical processes. Thus, the sequential organization of acoustic elements in two learned vocal communication signals (speech and birdsong) shows functionally equivalent dynamics, governed by similar processes.

Selective Formation of Porous Pt Nanorods for Highly Electrochemically Efficient Neural Electrode Interfaces.

The enhanced electrochemical activity of nanostructured materials is readily exploited in energy devices, but their utility in scalable and human-compatible implantable neural interfaces can significantly advance the performance of clinical and research electrodes. We utilize low-temperature selective dealloying to develop scalable and biocompatible one-dimensional platinum nanorod (PtNR) arrays that exhibit superb electrochemical properties at various length scales, stability, and biocompatibility for high performance neurotechnologies. PtNR arrays record brain activity with cellular resolution from the cortical surfaces in birds and nonhuman primates. Significantly, strong modulation of surface recorded single unit activity by auditory stimuli is demonstrated in European Starling birds as well as the modulation of local field potentials in the visual cortex by light stimuli in a nonhuman primate and responses to electrical stimulation in mice. PtNRs record behaviorally and physiologically relevant neuronal dynamics from the surface of the brain with high spatiotemporal resolution, which paves the way for less invasive brain-machine interfaces.

In Vivo Dopamine Detection and Single Unit Recordings Using Intracortical Glassy Carbon Microelectrode Arrays.

In this study, we present a 4-channel intracortical glassy carbon (GC) microelectrode array on a flexible substrate for the simultaneous in vivo neural activity recording and dopamine (DA) concentration measurement at four different brain locations (220µm vertical spacing). The ability of GC microelectrodes to detect DA was firstly assessed in vitro in phosphate-buffered saline solution and then validated in vivo measuring spontaneous DA concentration in the Striatum of European Starling songbird through fast scan cyclic voltammetry (FSCV). The capability of GC microelectrode arrays and commercial penetrating metal microelectrode arrays to record neural activity from the Caudomedial Neostriatum of European starling songbird was compared. Preliminary results demonstrated the ability of GC microelectrodes in detecting neurotransmitters release and recording neural activity in vivo. GC microelectrodes array may, therefore, offer a new opportunity to understand the intimate relations linking electrophysiological parameters with neurotransmitters release.

Stimulus bandwidth impact on auditory evoked potential thresholds and estimated upper-frequency limits of hearing in dolphins.

The frequency range of hearing is important for assessing the potential impact of anthropogenic noise on marine mammals. Auditory evoked potentials (AEPs) are commonly used to assess toothed whale hearing, but measurement methods vary across researchers and laboratories. In particular, estimates of the upper-frequency limit of hearing (UFL) can vary due to interactions between the unintended spread of spectral energy to frequencies below the desired test frequency and a sharp decline in hearing sensitivity at frequencies near the UFL. To assess the impact of stimulus bandwidth on UFL measurement, AEP hearing tests were conducted in four bottlenose dolphins (Tursiops truncatus) with normal and impaired hearing ranges. Dolphins were tested at frequencies near the UFL and at a frequency 1/2-octave below the UFL, where hearing sensitivity was better (i.e., threshold was lower). Thresholds were measured using sinusoidal amplitude modulated (SAM) tones and tone-bursts of varying bandwidth. Measured thresholds varied inversely as a function of stimulus bandwidth near the UFL with narrow-band tone-bursts approximating thresholds measured using SAM tones. Bandwidth did not impact measured thresholds where hearing was more sensitive, highlighting how stimulus bandwidth and the rate of decline of hearing sensitivity interact to affect measured threshold near the UFL.

Central auditory neurons have composite receptive fields.

High-level neurons processing complex, behaviorally relevant signals are sensitive to conjunctions of features. Characterizing the receptive fields of such neurons is difficult with standard statistical tools, however, and the principles governing their organization remain poorly understood. Here, we demonstrate multiple distinct receptive-field features in individual high-level auditory neurons in a songbird, European starling, in response to natural vocal signals (songs). We then show that receptive fields with similar characteristics can be reproduced by an unsupervised neural network trained to represent starling songs with a single learning rule that enforces sparseness and divisive normalization. We conclude that central auditory neurons have composite receptive fields that can arise through a combination of sparseness and normalization in neural circuits. Our results, along with descriptions of random, discontinuous receptive fields in the central olfactory neurons in mammals and insects, suggest general principles of neural computation across sensory systems and animal classes.

Songbirds use spectral shape, not pitch, for sound pattern recognition.

Humans easily recognize "transposed" musical melodies shifted up or down in log frequency. Surprisingly, songbirds seem to lack this capacity, although they can learn to recognize human melodies and use complex acoustic sequences for communication. Decades of research have led to the widespread belief that songbirds, unlike humans, are strongly biased to use absolute pitch (AP) in melody recognition. This work relies almost exclusively on acoustically simple stimuli that may belie sensitivities to more complex spectral features. Here, we investigate melody recognition in a species of songbird, the European Starling (Sturnus vulgaris), using tone sequences that vary in both pitch and timbre. We find that small manipulations altering either pitch or timbre independently can drive melody recognition to chance, suggesting that both percepts are poor descriptors of the perceptual cues used by birds for this task. Instead we show that melody recognition can generalize even in the absence of pitch, as long as the spectral shapes of the constituent tones are preserved. These results challenge conventional views regarding the use of pitch cues in nonhuman auditory sequence recognition.